Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TL/CUDA: Linear Broadcast for GPU #948

Open
wants to merge 35 commits into
base: master
Choose a base branch
from
Open

TL/CUDA: Linear Broadcast for GPU #948

wants to merge 35 commits into from
+608 −93

Conversation

ikryukov
Copy link
Contributor

@ikryukov ikryukov commented Mar 22, 2024 •
edited
Loading

What

Linear CUDA Broadcast implementation.

Why ?

  • Functional improvement, parity with others communication libraries.
  • Ability to place many ranks on single GPU
  • No GPU blocking, communication initiated from host

How ?

Naive approach where root rank writes data to own shared buffer and others ranks read from it through NVLink.

@ikryukov
Copy link
Contributor Author

ikryukov commented Mar 22, 2024 •
edited
Loading

Configuration string:
./configure --with-ucx=$HPCX_UCX_DIR --with-cuda=/usr/local/cuda --with-mpi=$HPCX_MPI_DIR --enable-gtest --prefix=$PWD/install --with-nvcc-gencode="-gencode=arch=compute_80,code=sm_80" --enable-debug
Run string:
mpirun --mca coll ^hcoll --mca coll_ucc_enable 0 -x LD_LIBRARY_PATH=/home/ikryukov/work/ucc/install/lib:$LD_LIBRARY_PATH -x UCC_CLS=basic -x UCC_TLS=ucp,cuda -x xUCC_LOG_LEVEL=info -x UCC_TL_CUDA_LOG_LEVEL=debug -x UCC_LOG_LEVEL=info -x UCC_CONFIG_FILE= -np 2 ./install/bin/ucc_test_mpi -c bcast --teams world -M cuda -O 0 -S 2

@ikryukov ikryukov marked this pull request as draft March 22, 2024 14:32
@swx-jenkins3
Copy link

Can one of the admins verify this patch?

@ikryukov ikryukov force-pushed the cuda_bcast branch 3 times, most recently from 00f3922 to 6caea67 Compare July 4, 2024 15:38
@ikryukov ikryukov marked this pull request as ready for review August 2, 2024 16:17
samnordmann
samnordmann reviewed Aug 19, 2024
View reviewed changes
Copy link
Collaborator

@samnordmann samnordmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks!
I only left some minor remarks. Can you, in addition, add this algo to the tests?

src/components/tl/cuda/tl_cuda.h Outdated Show resolved Hide resolved
src/components/tl/cuda/bcast/bcast_linear.c Outdated Show resolved Hide resolved
src/components/tl/cuda/bcast/bcast_linear.c Outdated Show resolved Hide resolved
src/components/tl/cuda/bcast/bcast_linear.c Outdated Show resolved Hide resolved
src/components/tl/cuda/bcast/bcast_linear.c Outdated Show resolved Hide resolved
src/components/tl/cuda/bcast/bcast_linear.c Show resolved Hide resolved
src/components/tl/cuda/bcast/bcast_linear.c
return;
}
} else {
ucc_debug("etask is nullptr");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this case an infinite loop? I am not sure to understand

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is error case, used it for debug, it should not happen in real situations

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, why not using ucc_assert here then?

src/components/tl/cuda/bcast/bcast_linear.c Outdated Show resolved Hide resolved
@ikryukov
Copy link
Contributor Author

Looks good to me! Thanks! I only left some minor remarks. Can you, in addition, add this algo to the tests?

Thanks for review, addressed comments and added test to validate bcast for cuda too.

Sergei-Lebedev
Sergei-Lebedev reviewed Aug 30, 2024
View reviewed changes
src/components/tl/cuda/bcast/bcast_linear.c Outdated Show resolved Hide resolved
src/components/tl/cuda/bcast/bcast_linear.c Show resolved Hide resolved
src/components/tl/cuda/tl_cuda.h Outdated Show resolved Hide resolved
@samnordmann samnordmann self-requested a review September 9, 2024 08:16
janjust
janjust approved these changes Sep 9, 2024
View reviewed changes
Copy link
Collaborator

@janjust janjust left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@manjugv
Copy link
Contributor

manjugv commented Sep 18, 2024

ping @Sergei-Lebedev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Sergei-Lebedev Sergei-Lebedev Sergei-Lebedev left review comments

@samnordmann samnordmann samnordmann approved these changes

@janjust janjust janjust approved these changes

Assignees
No one assigned
Labels
Ready-for-Review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@ikryukov @swx-jenkins3 @manjugv @janjust @Sergei-Lebedev @samnordmann